An Overview of Distance Metric Learning

In our previous comprehensive survey [41], we have categorized the disparate issues in distance metric learning. Within each of the four categories, we have summarized existing work, disclosed their essential connections, strengths and weaknesses. The first category is supervised distance metric learning, which contains supervised global distance metric learning, local adaptive supervised distance metric learning, Neighborhood Component Analysis (NCA) [13], and Relevant Components Analysis (RCA) [1]. The second category is unsupervised distance metric learning, covering linear (Principal Component Analysis (PCA) [14], Multidimensional Scaling (MDS) [5]) and nonlinear embedding methods (ISOMAP [35], Locally Linear Embedding (LLE) [30], and Laplacian Eigenamp (LE) [2]). We further unify these algorithms into a common framework based on the embedding computation. The third category, which is maximum margin based distance metric learning approaches, includes the large margin nearest neighbor based distance metric learning methods and semi-definite Programming (SDP) methods to solve the kernelized margin maximization problem. And the fourth category discussing kernel methods towards learning distance metrics, covers kernel alignment [28] and its SDP approaches [26], and also the extension work of learning the idealized kernel [25]. In addition to this survey [41], here we provide a complete and updated summarization of the related work on both unsupervised distance metric learning and supervised distance metric learning, including the most recent work in the area of distance metric learning. Many unsupervised distance metric learning algorithms are essentially for the purpose of unsupervised dimensionality reduction, i.e. learning a low-dimensional embedding of the original feature space. This group of methods can be divided into nonlinear and linear methods. The well known algorithms for nonlinear unsupervised dimensionality reduction are ISOMAP [35], Locally Linear Embedding (LLE) [30], and Laplacian Eigenamp (LE) [2]. ISOMAP seeks the subspace that best preserves the geodesic distances between any two data points, while LLE and LE focus on the preservation of local neighbor structure. An improved and stable version of LLE is achieved in [45], by introducing multiple linearly independent local weight vectors for each neighborhood. Among the linear methods, Principal Component Analysis (PCA) [14] finds the subspace that best preserves the variance of the data; Multidimensional Scaling (MDS) [5] finds the low-rank projection that best preserves the inter-point distance given by the pairwise distance matrix; Independent components analysis (ICA) [4] seeks a linear transformation to coordinates in which the data are maximally statistically indepen-